Migrate status & stats APIs to KP + remove legacy status lib #76054

joshdover · 2020-08-26T23:42:47Z

Summary

Related #41983

This migrates the /api/status and /api/stats endpoints to the Kibana Platform. It also deletes all of the src/legacy/server/status module and adapts any leftover code that was still using it. The updated status endpoint will now report statuses from all Kibana Platform plugins & core services.

Some discussion points:

While these two endpoints are very similar, /api/stats relies on UsageCollectors which are only available to plugins. Because of this, I opted to put the /api/stats endpoint in the usage_collection plugin while keeping the /api/status endpoint inside Core. I had to duplicate a bit of the logic for translating KP status levels to the legacy status colors. LMK if you have a better idea or prefer to put these elsewhere.
This introduces a new status format for /api/status which is disabled by default but can be opted-into with the v8format query parameter. My plan is to add a release note that the old format is deprecated and the default will change in 8.0/master. I will put up a quick follow up PR to change this in master after this merges.

Release Note

The /api/status endpoint response format is now deprecated and will change in 8.0.

This change only affects the status.* key of the JSON response, all other keys are unchanged:

Before

// curl mykibana.com/api/status
{
  "status": {
    "overall": {
      "state": "green",
      "title": "Green",
      "uiColor": "secondary",
      "icon": "success"
    },
    "statuses": [
      {
        "id": "elasticsearch@7.9.0",
        "state": "green",
        "message": "Ready",
        "uiColor": "secondary",
        "icon": "success"
      },
      {
        "id": "dashboard@7.9.0",
        "state": "green",
        "message": "Ready",
        "uiColor": "secondary",
        "icon": "success"
      },
      {
        "id": "visualize@7.9.0",
        "state": "green",
        "message": "Ready",
        "uiColor": "secondary",
        "icon": "success"
      },
      /** all other plugins */
    ]
  }
  /** all other fields are unchanged */
}

After

// curl mykibana.com/api/status
{
  "status": {
    "overall": {
      "level": "available",
      "summary": "All systems are ago!"
    },
    "core": {
      "elasticsearch": {
	    "level": "available",
	    "summary": "All systems are ago!"
	  },
      "savedObjects": {
	    "level": "available",
	    "summary": "All systems are ago!"
	  },
   },
   "plugins": {
     "dashboard": {
	    "level": "available",
	    "summary": "All systems are ago!"
	  },
     "visualize": {
	    "level": "available",
	    "summary": "All systems are ago!"
	  },
      /** ...all other plugins */
    }
  },
  /** all other fields are unchanged */
}

Status codes and request parameters are unchanged.

In all 7.x versions, the old format will continue to be returned. In 8.0, the default format will change to the new format. Users of 7.10+ may opt-in to the new format by specifying the v8format=true parameter in the URL. This allows admins to update any external health checks at your convenience and provide a smooth upgrade experience to 8.0.

Checklist

Delete any items that are not applicable to this PR.

Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

afharo

Just passing by because I've been working on elastic/beats#20956 and added some comments related to the /api/status API 🙂

src/core/server/status/routes/status.ts

src/plugins/usage_collection/server/plugin.ts

joshdover · 2020-09-10T21:29:03Z

src/plugins/usage_collection/server/routes/integration_tests/stats.test.ts

+type HttpService = ReturnType<typeof createHttpServer>;
+type HttpSetup = UnwrapPromise<ReturnType<HttpService['setup']>>;
+
+describe('/api/stats', () => {


The test coverage on this endpoint is admittedly low. Since this will be owned by @elastic/telemetry and they have more knowledge of usage collectors (which is the bulk of the logic in this endpoint), I'm leaving this to them to flesh out more. This simply serves as a smoke test to ensure the request will complete successfully. There are also existing integration tests that test more permutations of auth & query parameter combos.

Since this will be owned by @elastic/telemetry

Are you going to update CODEOWNERS in the following PR?

They already own this entire plugin, so I don't believe any changes are needed to CODEOWNERS?

kibanamachine · 2020-09-17T09:04:50Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 9f619ee

Build metrics

distributable file count

id	value	diff	baseline
default	45906	-12	45918
oss	27707	-11	27718
total		-23

History

💔 Build #75024 failed b327503
💔 Build #74983 failed 045f69a
💔 Build #74965 failed 88121e4
💚 Build #74833 succeeded a10e7e8
💔 Build #74641 failed 64576c2

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

afharo · 2020-09-17T11:03:14Z

That was it! Thank you for fixing it!

…#76054) * Complete migration of legacy status API * PR comments * Remove optional auth on stats endpoint * Regen docs * PR comments * PR nits * Set ReplaySubject buffer to 1 * Fix exclude_usage logic Co-authored-by: pgayvallet <pierre.gayvallet@elastic.co>

…#77752) * Complete migration of legacy status API * PR comments * Remove optional auth on stats endpoint * Regen docs * PR comments * PR nits * Set ReplaySubject buffer to 1 * Fix exclude_usage logic Co-authored-by: pgayvallet <pierre.gayvallet@elastic.co> Co-authored-by: Josh Dover <me@joshdover.com>

…rok/new-patterns-component-use-array * 'master' of github.com:elastic/kibana: (140 commits) Add telemetry as an automatic privilege grant (elastic#77390) [Security Solutions][Cases] Cases Redesign (elastic#73247) Use Search API in TSVB (elastic#76274) [Mappings editor] Add support for constant_keyword field type (elastic#76564) [ML] Adds ML modules for Metrics UI Integration (elastic#76460) [Drilldowns] {{event.points}} in URL drilldown for VALUE_CLICK_TRIGGER (elastic#76771) Migrate status & stats APIs to KP + remove legacy status lib (elastic#76054) use App updater API instead of deprecated chrome.navLinks.update (elastic#77708) [CSM Dashboard] Remove points from line chart (elastic#77617) [APM] Trace timeline: Replace multi-fold function icons with new EuiIcon glyphs (elastic#77470) [Observability] Overview: Alerts section style improvements (elastic#77670) Bump the Node.js version used by Docker in CI (elastic#77714) Upgrade all minimist (sub)dependencies to version ^1.2.5 (elastic#60284) Remove unneeded forced package resolutions (elastic#77467) [ML] Add metrics app to check made for internal custom URLs (elastic#77627) Functional tests - add supertest for test_user (elastic#77584) [ML] Adding option to create AD jobs without starting the datafeed (elastic#77484) Bump node-fetch to 2.6.1 (elastic#77445) Bump sharkdown from v0.1.0 to v0.1.1 (elastic#77607) [APM]fixing y axis on transaction error rate to 100% (elastic#77609) ... # Conflicts: # x-pack/plugins/ingest_pipelines/public/application/components/pipeline_processors_editor/components/manage_processor_form/manage_processor_form.container.tsx # x-pack/plugins/ingest_pipelines/public/application/components/pipeline_processors_editor/components/manage_processor_form/manage_processor_form.tsx # x-pack/plugins/ingest_pipelines/public/application/components/pipeline_processors_editor/components/processor_form/field_components/drag_and_drop_text_list.scss # x-pack/plugins/ingest_pipelines/public/application/components/pipeline_processors_editor/components/processor_form/field_components/drag_and_drop_text_list.tsx # x-pack/plugins/ingest_pipelines/public/application/components/pipeline_processors_editor/components/processor_form/field_components/text_editor.scss # x-pack/plugins/ingest_pipelines/public/application/components/pipeline_processors_editor/components/processor_form/processors/grok.test.tsx

joshdover force-pushed the status/api-migration branch 9 times, most recently from 93ef427 to 76c334e Compare September 2, 2020 23:40

joshdover changed the title ~~[wip] Migrate /api/status to KP~~ [wip] Complete migration of legacy status API Sep 2, 2020

afharo mentioned this pull request Sep 3, 2020

[Metricbeat][Kibana][status] metrics is optional elastic/beats#20956

Closed

6 tasks

afharo reviewed Sep 3, 2020

View reviewed changes

src/core/server/status/routes/status.ts Show resolved Hide resolved

src/core/server/status/routes/status.ts Outdated Show resolved Hide resolved

joshdover force-pushed the status/api-migration branch 2 times, most recently from c4e79ef to 0713257 Compare September 6, 2020 18:31

thbkrkr mentioned this pull request Sep 9, 2020

Avoid panic on Kibana telemetry API call elastic/cloud-on-k8s#3736

Merged

joshdover force-pushed the status/api-migration branch 7 times, most recently from 00ffe99 to 47354d0 Compare September 10, 2020 18:37

Complete migration of legacy status API

499868c

joshdover force-pushed the status/api-migration branch from 47354d0 to 499868c Compare September 10, 2020 21:19

joshdover commented Sep 10, 2020

View reviewed changes

src/plugins/usage_collection/server/plugin.ts Outdated Show resolved Hide resolved

joshdover commented Sep 10, 2020

View reviewed changes

joshdover added Feature:Legacy Removal Issues related to removing legacy Kibana release_note:deprecation Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v7.10.0 labels Sep 10, 2020

Merge remote-tracking branch 'upstream/master' into status/api-migration

b327503

pgayvallet mentioned this pull request Sep 17, 2020

remove legacy ES plugin #77703

Merged

Merge remote-tracking branch 'upstream/master' into status/api-migration

9f619ee

pgayvallet merged commit cbf2844 into elastic:master Sep 17, 2020

pgayvallet mentioned this pull request Sep 17, 2020

[7.x] Migrate status & stats APIs to KP + remove legacy status lib (#76054) #77752

Merged

joshdover deleted the status/api-migration branch September 22, 2020 20:21

mindw mentioned this pull request Jan 3, 2021

Kibana replies with 400 chamilad/kibana-prometheus-exporter#7

Closed

joshdover mentioned this pull request Mar 11, 2021

Remove legacy /api/status endpoint response format #94419

Closed

afharo mentioned this pull request Mar 25, 2021

[test-failed]: X-Pack API Integration Tests1.x-pack/test/api_integration/apis/kibana/stats/stats·js - apis kibana stats /api/stats operational stats and usage stats no auth should return 200 and stats for no extended #51491

Closed

stefnestor mentioned this pull request Apr 9, 2022

Add Kibana Status API elastic/support-diagnostics#584

Merged

afharo mentioned this pull request Jul 17, 2024

[Metricbeat/kibana/status] Add support for v8format elastic/beats#40275

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate status & stats APIs to KP + remove legacy status lib #76054

Migrate status & stats APIs to KP + remove legacy status lib #76054

joshdover commented Aug 26, 2020 •

edited

Loading

afharo left a comment

joshdover Sep 10, 2020 •

edited

Loading

mshustov Sep 14, 2020

joshdover Sep 14, 2020

kibanamachine commented Sep 17, 2020

afharo commented Sep 17, 2020

Migrate status & stats APIs to KP + remove legacy status lib #76054

Migrate status & stats APIs to KP + remove legacy status lib #76054

Conversation

joshdover commented Aug 26, 2020 • edited Loading

Summary

Release Note

Before

After

Checklist

For maintainers

afharo left a comment

Choose a reason for hiding this comment

joshdover Sep 10, 2020 • edited Loading

Choose a reason for hiding this comment

mshustov Sep 14, 2020

Choose a reason for hiding this comment

joshdover Sep 14, 2020

Choose a reason for hiding this comment

kibanamachine commented Sep 17, 2020

💚 Build Succeeded

Build metrics

distributable file count

History

afharo commented Sep 17, 2020

joshdover commented Aug 26, 2020 •

edited

Loading

joshdover Sep 10, 2020 •

edited

Loading